Training and selecting the best model should be automated and run at some frequency that meets the needs of your application, but is often on the order of hours. It may be run using a job scheduling service, or even as a streaming job with a long update interval. In the case of Oryx, the offline training is implemented as an operation on a Spark DStream of new user/item interaction data. The new data that has arrived within the latest batch interval is combined with historical data from the Hadoop filesystem (HDFS) and used as the training data for the Spark ML alternating least squares (ALS) algorithm.

A recent example of such work is the ICLR 2016 paper "Learning to Diagnose with LSTM Recurrent Neural Networks" (of which Mr. Kale is a joint first author in his capacity as a PhD candidate at the USC Information Science Institute). In it, the authors trained a LSTM RNN or LSTM, to classify acute care diseases such as respiratory distress in critically ill children. The RNN (and the more complex LSTM RNN) is a neural net architecture with recurrent connections between hidden states, giving it a form of persistent state (or "memory") across sequential inputs. These connections enable RNNs to detect relationships not only between inputs, e.g., heart rate and blood pressure, but also over time, e.g., between a patient's state at time of admission and later in an ICU stay. This makes it especially well-suited to health problems, which often involve modeling changes over time.

We can also independently test the model performance on a test set using any of the trained model snapshots saved at the checkpoint location. If ever the model performance improves initially and then starts to flatten or decrease it might be a good idea to reduce the learning rate at that point while resuming training from where it left off. All one would need to do is use the model snapshot from the 15th epoch, which would be a minor change to the code above.

Data scientists have hundreds of probability distributions from which to choose. Data science, whatever it may be, remains a big deal. "A data scientist is better at statistics than any software engineer," you may overhear a pundit say, at your local tech get-togethers and hackathons. The applied mathematicians have their revenge, because statistics hasn't been this talked-about since the roaring 20s. They have their own legitimizing Venn diagram of which people don't make fun. Suddenly it's you, the engineer, left out of the chat about confidence intervals instead of tutting at the analysts who have never heard of the Apache Bikeshed project for distributed comment formatting. To fit in, to be the life and soul of that party again, you need a crash course in stats.

Intel MKL is a library of optimized math routines that are hand-optimized specifically for Intel processors. For example, it includes highly-optimized routines for Linear Algebra, Fast Fourier Transforms (FFT), Vector Math and Statistics functions. These mathematical operations are building blocks for machine learning and related analytic algorithms, and thus integration with MKL delivers massive performance boost for machine learning workloads. Spark is already instrumented to take advantage of optimized implementations of these routines using netlib-java, but still requires the addition of an implementation like MKL to activate these optimizations.

Thanks to Michal Malohlava, Amy Wang, and Avni Wadhwa of H20.ai for providing the following guest post about building ML apps using Sparkling Water and Apache Spark on CDH. The Sparkling Water project is nearing its one-year anniversary, which means Michal Malohlava, our main contributor, has been very busy for the better part of this past year. The Sparkling Water project combines H2O machine-learning algorithms with the execution power of Apache Spark. This means that the project is heavily dependent on two of the fastest growing machine-learning open source projects out there. With every major release of Spark or H2O there are API changes and, less frequently, major data structure changes that affect Sparkling Water.

Creating and training machine-learning models is more complex on distributed systems, but there are lots of frameworks for abstracting that complexity. There are more options now than ever from proven open source projects for doing distributed analytics, with Python and R become increasingly popular. In this post, you'll learn the options for setting up a simple read-eval-print (REPL) environment with Python and R within the Cloudera QuickStart VM using APIs for two of the most popular cluster computing frameworks: Apache Spark (with MLlib) and H2O (from the company with the same name). To compare these approaches, you'll train a linear regression against a data set with known coefficients. Spark includes PySpark (supported by Cloudera), the Python API for Spark.

Under reasonable circumstances, how much money can you expect to lose? The financial statistic value at risk (VaR) seeks to answer this question. Since its development on Wall Street soon after the stock market crash of 1987, VaR has been widely adopted across the financial services industry. Some organizations report the statistic to satisfy regulations, some use it to better understand the risk characteristics of large portfolios, and others compute it before executing trades to help make informed and immediate decisions. For reasons that we will delve into later, reaching an accurate estimate of VaR can be a computationally expensive process.